Surface contextual bandits, and match trackingCallback params to the SDK version - #113
Merged
Merged
Conversation
The SDK types allow both `(experiment, result)` and `(experiment, result, userContext)` tracking callbacks, but DevTools flagged anything other than exactly 2 params as an implementation problem. Treat 2 or 3 params as valid, and stay quiet instead of warning when the param list can't be determined at all. Also: - Share the logic between the SDK tab, the panel copy, and the background icon status so they can't drift. - Replace the `/\(([^)]+)\)/` param scrape, which truncated at destructured params and default values containing parens, and mis-parsed single-param arrow functions. - Forward extra args through the patched trackingCallback and onFeatureUsage wrappers so a userContext arg isn't swallowed.
DevTools derives its "latest SDK version" from the bundled SDK's package.json, so a stale dependency means stale outdated-version warnings. 1.7.0 also passes a userContext arg to instance-level trackingCallbacks, which the arg forwarding in the previous commit now preserves.
The row now reads "Found (3 params)" and the panel names the detected signature, so the callback shape is visible without opening the page's source.
Self-contained page for exercising the SDK tab with no GrowthBook account: apiHost points at the same static server, and a static api/features/local-test payload satisfies the canConnect probe. ?params=1|2|3|4 switches the trackingCallback arity and &ofu=1 supplies an onFeatureUsage callback. The SDK bundle is copied in by `yarn test-page` rather than vendored.
This reverts commit dd4b2e5.
onFeatureUsage only ever lands in the SDK's user context, which core.ts calls as cb(key, result); the 3-arg form is GrowthBookClient's global context, which DevTools never patches. So the rest param was always empty and the cast only silenced a type error for an argument that cannot arrive. trackingCallback genuinely does receive a third userContext arg in 1.7+, so that forwarding stays - but now via a typed optional param rather than a rest spread plus a cast.
- deferred tracking calls kept only {experiment, result}, but
fireDeferredTrackingCalls replays them as
trackingCallback(experiment, result, call.user), so a replayed
3-param callback got user === undefined
- the wrapper discarded the callback's return value, so the SDK could
no longer await an async trackingCallback
- a zero-arity forwarding wrapper parsed to [] and warned; that shape is
common in minified analytics code and works fine
- parseCallbackParams desynced on brackets and commas inside string
defaults, eg (experiment, result, sep = ",")
- the toString guard could throw a TypeError out of the injected script
when a page shadowed toString to return a non-string
- the row read "Found (1 params)" and counted rest params, disagreeing
with the detail panel
Also makes the bare-arrow test exercise the branch it names - a type
annotation forced parens, so deleting the branch left the suite green.
- parseCallbackParams no longer wraps toString in try/catch with a typeof guard; main never did and callers pass a real function - trim multi-line comments down to one line each - flatten the nested ternary behind the SDK tab row label - straighten out the trackingCallback panel branches so each renders a whole sentence instead of splicing a conditional before the period - drop the any casts from the tests
1.7.0 is where the SDK started calling the instance-level callback as (experiment, result, userContext); through 1.6.5 it only ever passed two args. So the expected arity is exact in both directions - two params on 1.7.0+ silently drops the userContext payload, and three on anything older leaves that param permanently undefined. Both now read as issues, with a panel message naming the version and what is lost. Stays lenient when the page's SDK version is unknown.
The third param is not what carries the user's attributes - omitting it just means missing newer features like contextual bandits.
Contextual bandit rules keep their variations under contextualVariations so pre-1.7 SDKs skip them, but getFeatureExperiments gated on rule.variations - so they never reached the Experiments tab at all. Read either field, then mark and explain the ones that are bandits. - "Contextual Bandit" badge on the experiment row, the detail header, and matching exposure entries in Event Logs - detail section with the weights actually applied to this user, as percentages to match Rule.tsx, and the attributes the matched context tested - both collapsible, since a bandit can have many of either - calls out the three states that otherwise look like a healthy bandit: ref missing from the payload, no matching context, and not bucketed Weights come from the feature's experimentResult: the SDK applies bandit logic while evaluating the rule, so gb.run() and the raw rule never carry them.
A live payload marks a contextual bandit rule with contextualVariations alone: contextualBanditRef and the top-level contextualBandits block only appear once the bandit has trained contexts. Keying detection off the ref meant a real bandit rendered as an ordinary experiment. Detection now keys off contextualVariations, and the detail panel handles a bandit with no trained contexts by showing the rule's fallback weights and saying that is what every user gets.
It sat just above Implementation, well below the fold, so a bandit read as an ordinary experiment on open. It now follows Current value, and its heading matches the label style of the sections around it.
- lead with a meta box: bandit ref, current leaf, bandit version - "Variation Weights" replaces the fallback wording, and the assigned variation is emphasised in the bar list - hide Targeting and Traffic for bandits, whose static rule weights contradict the dynamic ones shown above - render context values directly rather than through ValueField
- ref, current leaf and version as a compact sub-line under the heading - leaf shown as a pill beside the weights label when a context matched - Setup block with the mockup's checks: whether trackingCallback receives userContext (bandit rewards can't be attributed without it) and whether a bandit definition is in the payload - weights read "Weights for this context" when a context matched, and "Variation Weights" otherwise
- drop the unused getContextualBandit export and un-export the types and condition walker that nothing outside the module names - cut multi-line explanatory comments down to one line each - use clsx for the conditional weight-row classes, as elsewhere in the app - cover the condition walker through getMatchedContextAttributes now that it is private
Contributor
Author
Result.key is the variation key (meta.key, or the variation index), not the experiment key, so matching it against experiment.key found nothing: the panel always reported no trained contexts and fell back to the rule's static weights. Match on the feature result's experiment key instead, and only build the context object when bandit weights are actually present.
- evaluate bandit experiments from the feature result rather than run(), which skips the feature-rule path and so reports a variation the page is not serving - read that result from the parent instead of mounting a third sandbox evaluation in the detail panel - detect bandits in the logs via the runtime experiment's contextualBandit; the ref and contextualVariations are stripped by the time a log is built - reuse ContextualBanditBadge in the experiments list and logs - stop reporting an untrained bandit as a warning, and stop blaming the callback for a userContext an older SDK never passes - say "Not applied for this user" when a definition exists but no bandit weights were used, rather than claiming the payload has none
For a bandit with no trained contexts the sub-line read "leaf: No trained contexts yet", labelling a leaf that does not exist and repeating what the setup check says. The leaf now only appears once weights were applied, and a definition that exists but was not used is called out in the checks.
Radio cards stop being scannable once an experiment has more than a handful of variations, which a bandit routinely does. Also say what to do when a forced variation is hiding the bandit weights: forcing sets hashUsed false, and the SDK then drops leafId and variationWeights entirely.
The dropdown used position="popper", unlike every other Select in the app; inside the fixed, scrolling detail panel its menu did not open. Match the existing variant="soft" usage. The leaf was buried in the grey sub-line and hard to find, so it now has its own labelled row.
Without a value for the rule's hashAttribute the SDK logs "Skip because missing hashAttribute" and returns no result at all, so the panel said only that weights were not applied. It now names the attribute.
Selecting by value reads better than by index when the variations are plain strings, as a bandit's usually are. Object-valued variations keep the syntax-highlighted display, where a dropdown of JSON blobs would be unreadable.
…the panel Current value goes back to the read-only value display. The variation dropdown rendered but would not open: the detail panel sets z-index 1000 inline, and the portaled menu landed underneath it. Give the menu content a higher z-index and keep popper positioning, which the nav select already uses.
useGBSandboxEval stuffs rule metadata for its debug log, but when a rule had no meta it substituted a single-element array however many variations the rule had. The SDK indexes experiment.meta by variation index, so getExperimentResult threw on any variation past the first, the whole evaluate() call died, and every experiment then rendered as Inactive with a null value - the bandit panel included. Build the stuffed meta per variation, keeping any entries the rule already had, and keep the single tag for rules with no variations since the debug log reads meta[0].
Three attempts at making a Radix Select open inside the detail panel failed. The app already has SelectField, which portals its menu through SelectMenuPortalProvider - the component that exists precisely because menus in these panels need an explicit portal target. Use it, keeping the variation icons via formatOptionLabel and sort disabled so the options stay in variation order. Also hide the debug box under Enrollment Status when there is no message; lastDebugLog defaults to an empty string, which rendered an empty console panel.
…g lines SelectField labels the selected value with the raw value, so the closed dropdown read "1" rather than the variation name. Derive the name from the index in formatOptionLabel, which covers the trigger and the menu. Also stop echoing "In experiment" and "Force via dev tools" in the debug box: the panel states both itself, right above it.
Both sections used a plain Text heading with the caret, which read as a title rather than a toggle and sat tight against the row above. Use the bold Link trigger and accordion my-4 spacing that SdkItemPanel already uses for its collapsible sections.
Both accordion bodies started flush against the toggle.
Reverts the content margin from 180abd8: the gap belongs between the previous section and the toggle, so each trigger now carries the top margin.
- Rule.tsx destructured rule.variations, so a bandit rule rendered in the Features tab as a rollout with no variations. Use ruleVariations, the same fix already applied to getFeatureExperiments. - Don't highlight variation 0 for a user the SDK skipped: it clamps the index to 0 with inExperiment false, so the weights list contradicted the Inactive status shown above it. - The missing-hash-attribute diagnostic sat behind , which the common bandit shape never has, so it never fired. Hoist it, and count fallbackAttribute as satisfying the hash value. - Omit isContextualBandit when false: useSearch matches JSON.stringify(item), so a literal false made every experiment log row match a search for "bandit". - Reserve room for the badge in split view, where the types container is absolutely positioned and the name ran underneath it.
Bandit experiments had an empty Results log: `banditResult ?? run()` short-circuited run(), which is what populates the debug log. Always run, and prefer the bandit result afterwards. Remove the amber forced-variation warning, gated on `isForced && cb`, which can never both be true - forcing sets hashUsed false and the SDK then drops the weights that produce cb. The setup check already covers that case with something actionable. Gate the panel on the page's SDK version. DevTools evaluates with its own bundled SDK, so on a pre-1.7 page it would report a bandit assignment the page never made; say so instead.
bryce-fitzsimons
approved these changes
Sep 2, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Two related pieces of work, both needing SDK 1.7.0 (bumped here).
DevTools flagged any
trackingCallbackthat didn't take exactly 2 params asbroken. 1.7.0 passes a third
userContextarg, so the expected arity nowdepends on the page's SDK version. Contextual bandits, new in 1.7.0, were
invisible in DevTools entirely: their rules carry variations under
contextualVariationsso pre-1.7 SDKs skip them, and the Experiments tab onlylooked at
variations.Changes
Tracking callback
wrong-arity cases read as issues, with a panel message naming the version.
Found (3 params))./\(([^)]+)\)/param scrape, which truncated on destructuredparams and string defaults and misread single-param arrow functions.
trackingCallback, and keep it ondeferred tracking calls —
fireDeferredTrackingCallsreplays them withcall.user.Contextual bandits
contextualVariations ?? variationsingetFeatureExperiments, sobandits appear in the Experiments tab at all.
contextualVariations;contextualBanditRefand the payload'scontextualBanditsblock only appear once a bandit has trained contexts.Contextual Banditbadge on the experiment row, detail header, and matchingEvent Logs entries.
attributes the matched context tested, and setup checks. Weights and context
collapse, since a bandit can have many of either.
definition missing from the payload, no matching context, not bucketed, and
a forced variation or missing hash attribute suppressing the weights.
run(),which skips the feature-rule path and so reported a variation the page
wasn't serving.
the dynamic ones.
Fixes found along the way
useGBSandboxEvalstuffed a single-elementmetaarray into rules that hadnone, however many variations they had. The SDK indexes
experiment.metabyvariation, so
getExperimentResultthrew, the whole evaluation died, andevery experiment rendered as Inactive with a null value. Pre-existing and
unrelated to bandits — one meta-less rule anywhere took down the tab.
SelectFieldso the menu portals correctly inside the detail panel.In experimentandForce via dev toolsin the debug box, andhide it when empty.
Testing
yarn test— 52 unit tests covering param parsing, version-dependent arity,bandit detection, and the stuffed-meta length.
realPayload.test.tspins therule shape a live payload actually serves.
dist/unpacked against a site on SDK 1.7.0. SDK tab → TrackingCallback shows the param count; a 2-param callback reads as an issue.
Bandit section in the detail. With no trained contexts in the payload, expect
the rule's variation weights and no context section.
Dependencies
None.


